Imaging controller and imaging control method and program

ABSTRACT

An imaging controller includes an index calculator to calculate an index value for each of divided areas of images captured by a plurality of imaging units, the index value for evaluating a photographic state of each of the divided areas, an evaluation value calculator to evaluate the images and an overlapping area between the images on the basis of the index value of each divided area calculated by the index calculator and calculate an overall evaluation value, and a condition determiner to determine an imaging condition for each of the imaging units on the basis of the overall evaluation value calculated by the evaluation value calculator.

CROSS REFERENCE TO RELATED APPLICATION

The present application is based on and claims priority from JapanesePatent Application No. 2012-199622, filed on Sep. 11, 2012.

TECHNICAL FIELD

The present invention relates to an imaging controller which can provideappropriate imaging conditions to multiple imaging units, and an imagingcontrol method executed by the imaging controller and a program forrealizing the imaging control method.

BACKGROUND ART

There is a known omnidirectional imaging system which includes multiplewide-angle lenses such as fisheye lens or super wide-angle lens tocapture an image in omnidirections at once. It is configured to projectimages from the lenses onto a sensor surface and combine the imagesthrough image processing to thereby generate an omnidirectional image.For example, by use of two wide-angle lenses with angle of view of over180 degrees, omnidirectional images can be generated. In the imageprocessing a partial image captured by each lens system is subjected todistortion correction and projection conversion on the basis of acertain projection model with a distortion from an ideal model takeninto account. Then, the partial images are connected on the basis of anoverlapping portion of the partial images to form a singleomnidirectional image.

In related art an exposure correction technique of a digital camera toacquire a proper exposure from a captured image is known. For instance,Japanese Patent Application Publication No. 2007-329555 discloses animaging system including multiple imaging units arranged to have anoverlapping imaging area to extract an overlapping area from each of theimages captured by the imaging units. It is configured to adjust atleast one of the exposure and white balance of the imaging unitsaccording to each image of the extracted overlapping areas to reduce adifference in the brightness or color of the captured images, for thepurpose of abating workloads of post-processing such as synthesis.

However, it is difficult for the omnidirectional imaging system toacquire a proper exposure by such a related-art exposure correctiontechnique because the optical conditions or photographic circumstancesof the imaging units thereof differ. The related art disclosed in theabove document only concerns an overlapping area so that it cannotobtain appropriate exposure correction values under an unbalancedexposure condition if the overlapping area is small relative to theentire image. In particular, since the imaging area of theomnidirectional imaging system is omnidirectional, a high-brightnesssubject as the sun is often captured on a sensor, which may cause aflare and an increase in image offset value. A proper exposure can beobtained for each sensor, however, it may cause a difference inbrightness between the connecting portions of the images and impair thequality of an omnidirectional image.

DISCLOSURE OF THE INVENTION

The present invention aims to provide an imaging controller and imagingcontrol method and program which can provide to each of imaging units aproper imaging condition to abate a discontinuity at the connectingpoints of the images captured by the imaging units in synthesizing theimages.

According to one aspect of the present invention, an imaging controllercomprises an index calculator to calculate an index value for each ofdivided areas of images captured by a plurality of imaging units, theindex value for evaluating a photographic state of each of the dividedareas, an evaluation value calculator to evaluate the images and anoverlapping area between the images on the basis of the index value ofeach divided area calculated by the index calculator and calculate anoverall evaluation value, and a condition determiner to determine animaging condition for each of the imaging units on the basis of theoverall evaluation value calculated by the evaluation value calculator.

BRIEF DESCRIPTION OF THE DRAWINGS

Features, embodiments, and advantages of the present invention willbecome apparent from the following detailed description with referenceto the accompanying drawings:

FIG. 1 is a cross section view of an omnidirectional imaging systemaccording to the present embodiment;

FIG. 2 shows the hardware configuration of the omnidirectional imagingsystem in FIG. 1;

FIG. 3 shows a flow of the entire image processing of theomnidirectional imaging system in FIG. 1;

FIGS. 4A, 4B show 0^(th) and 1^(st) images captured by two fisheyelenses, respectively and FIG. 4C shows a synthetic image of the 0^(th)and 1^(st) captured images by way of example;

FIG. 5A, 5B show an area division method according to the presentembodiment;

FIG. 6 is a flowchart for exposure control executed by theomnidirectional imaging system according to the present embodiment; and

FIG. 7 is a flowchart for exposure calculation executed by theomnidirectional imaging system according to the present embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of an imaging controller and an imagingsystem will be described in detail with reference to the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts. By way ofexample, the present embodiment describes an omnidirectional imagingsystem which comprises a camera unit including two fisheye lenses and afunction to decide an imaging condition on the basis of images capturedby the two fisheye lenses. However, the present embodiment should not belimited to such an example. Alternatively, the omnidirectional imagingsystem can comprise a camera unit including three or more fisheye lensesto determine an imaging condition according to the images captured bythe fisheye lenses. Herein, a fisheye lens can include a wide-angle lensor a super wide-angle lens.

Referring to FIGS. 1 to 2, the overall configuration of anomnidirectional imaging system 10 is described. FIG. 1 is a crosssection view of the omnidirectional imaging system 10 (hereinafter,simply imaging system). It comprises a camera unit 12, a housing 14accommodating the camera unit 12 and elements as controller, batteries,and a shutter button 18 provided on the housing 14.

The camera unit 12 in FIG. 1 comprises two lens systems 20A, 20B and twosolid-state image sensors 22A, 22B as CCD (charge coupled device) sensoror CMOS (complementary metal oxide semiconductor). Herein, each of thepairs of the lens systems 20 and solid-state image sensors 22 arereferred to as imaging unit. The lens systems 20A, 20B are eachcomprised of 6 groups of 7 lenses as a fisheye lens, for instance. Inthe present embodiment the fisheye lens has total angle of view of 180degrees (360 degrees/n, n=2) or more, preferably 185 degrees or more,more preferably 190 degrees or more.

The optical elements as lenses, prisms, filters, aperture stops of thelens systems 20A, 20B are positioned relative to the solid-state imagesensors 22A, 22B so that the optical axes of the optical elements areorthogonal to the centers of the light receiving areas of thecorresponding solid-state image sensors 22 as well as the lightreceiving areas become the imaging planes of the corresponding fisheyelenses. The solid-state image sensors 22 are area image sensors on whichphotodiodes are two-dimensionally arranged, to convert light gathered bythe lens systems 20 to image signals.

In the present embodiment the lens systems 20A, 20B are the same anddisposed opposite to each other so that their optical axes coincide. Thesolid-state image sensors 22A, 22B convert light distribution to imagesignals and output them to a not-shown image processor on thecontroller. The image processor combines partial images from thesolid-state image sensors 22A, 22B to generate a synthetic image withsolid angle of 4π in radian or an omnidirectional image. Theomnidirectional image is captured in all the directions which can beseen from a shooting point. Instead of the omnidirectional image, apanorama image which is captured in a 360-degree range only on ahorizontal plane can be generated.

To form an omnidirectional image with use of the fisheye lenses withtotal angle of view of more than 180 degrees, an overlapping portion ofthe captured images by the imaging units is used for connecting imagesas reference data representing the same image. Generated omnidirectionalimages are output to, for instance, a display provided in or connectedto the camera unit 12, a printer or an external storage medium such asSD Card®, compact Flash®.

FIG. 2 shows the structure of hardware of the imaging system 10according to the present embodiment. The imaging system 10 comprises adigital still camera processor 100 (hereinafter, simply processor), alens barrel unit 102, and various elements connected with the processor100. The lens barrel unit 102 includes the two pairs of lens systems20A, 20B and solid-state image sensors 22A, 22B. The solid-state imagesensors 22A, 22B are controlled by a command from a CPU 130 of theprocessor 100.

The processor 100 comprises ISPs (image signal processors) 108A, 108B, aDMAC (direct memory access controller) 110, an arbiter (ARBMEMC) 112 formemory access, a MEMC (memory controller) 114 for memory access, and adistortion correction and image synthesis block 118. The ISPs 108A, 108Bperform automatic exposure control to and set white balance and gammabalance of image data signal-processed by the solid-state image sensors22A, 22B.

The MEMC 114 is connected to an SDRAM 116 which temporarily stores dataused in the processing of the ISP 108A, 108B and distortion correctionand image synthesis block 118. The distortion correction and imagesynthesis block 118 performs distortion correction and verticalinclination correction on the two partial images from the two imagingunits on the basis of information from a triaxial acceleration sensor120 and synthesizes them.

The processor 100 further comprises a DMAC 122, an image processingblock 124, a CPU 130, an image data transferrer 126, an SDRAMC 128, amemory card control block 140, a USB block 146, a peripheral block 150,an audio unit 152, a serial block 158, an LCD (Liquid Crystal Display)driver 162, and a bridge 168.

The CPU 130 controls the operations of the elements of the imagingsystem 10. The image processing block 124 performs various kinds ofimage processing on image data together with a resize block 132, a JPEGblock 134, and an H. 264 block 136. The resize block 132 enlarges orshrinks the size of image data by interpolation. The JPEG block 134 is acodec block to compress and decompress image data in JPEG. The H. 264block 136 is a codec block to compress and decompress video data inH.264. The image data transferrer 126 transfers the images processed bythe image processing block, 124. The SDRAMC 128 controls the SDRAM 138connected to the processor 100 and temporarily storing image data duringimage processing by the processor 100.

The memory card control block 140 controls data read and write to amemory card and a flash ROM 144 inserted to a memory card throttle 142in which a memory card is detachably inserted. The USB block 146controls USB communication with an external device such as personalcomputer connected via a USB connector 148. The peripheral block 150 isconnected to a power switch 166.

The audio unit 152 is connected to a microphone 156 for receiving anaudio signal from a user and a speaker 154 for outputting the audiosignal, to control audio input and output. The serial block 158 controlsserial communication with the external device and is connected to awireless NIC (network interface card) 160. The LCD driver 162 is a drivecircuit for the LCD 164 and converts the image data to signals fordisplaying various kinds of information on an LCD 164.

The flash ROM 144 contains a control program written in readable codesby the CPU 130 and various kinds of parameters. Upon power-on of thepower switch 166, the control program is loaded onto a main memory. TheCPU 130 controls the operations of the units and elements of the imageprocessor in compliance with the control program on the main memory, andtemporarily stores necessary control data in the SDRAM 138 and anot-shown local SRAM.

FIG. 3 shows essential function blocks for controlling imaging conditionand the flow of the entire image processing of the imaging system 10according to the present embodiment. First, the solid-state imagesensors 22A, 22B capture images under a certain exposure condition andoutput them. Then, the ISPs 108A, 108B in FIG. 2 perform optical blackcorrection, defective pixel correction, linear correction, shadingcorrection and area division (collectively referred to as firstprocessing) to the images from the solid-state image sensors 22A, 22Band store them in memory.

The optical black correction is a processing in which an output signalfrom an effective pixel area is subjected to clamp correction, using theoutput signals of optical black areas of the solid-state image sensorsas a black reference level. A solid-state image sensor such as CMOS maycontain defective pixels from which pixels values are not obtainablebecause of impurities entering a semiconductor substrate in themanufacturing of the image sensor. The defective pixel correction is aprocessing in which the value of a defective pixel is correctedaccording to a combined signal from neighboring pixels of the defectivepixel.

The linear correction is for each of RGBs. The shading correction is tocorrect a distortion of shading in an effective pixel area bymultiplying the output signal of the effective pixel area by a certaincorrection coefficient. The area division is to divide a captured imageinto small areas and calculate an integrated value or an integratedaverage value of brightness values for each divided area.

Returning to FIG. 3, after the first processing the ISPs 108A, 108Bfurther perform white balance, gamma correction, Bayer interpolation,YUV conversion, edge enhancement and color correction (collectivelyreferred to as second processing) to the images, and the images arestored in the memory. The amount of light transmitting through the colorfilters of the image sensors changes depending on the color of thefilter. The white balance correction is to correct a difference insensitivity to the three colors R (red), G (green), and B (blue) and seta gain for appropriately representing white color in an image. A WB(white balance) calculator 220 calculates a white balance parameteraccording to the RGB integrated value or integrated average valuecalculated in the area division process. The gamma correction is tocorrect a gamma value of an input signal so that the output linearity ofan output device is maintained with the characteristic thereof takeninto account.

Further, in the CMOS each pixel is attached with any of RGB colorfilters. The Bayer interpolation is to interpolate insufficient twocolors from neighboring pixels. The YUV conversion is to convert RAWdata in RGB format to data in YUV format of a brightness signal Y and acolor difference signal UV. The edge enhancement is to extract the edgesof an image according to a brightness signal, apply a gain to the edges,and remove noise in the image in parallel to the edge extraction. Thecolor correction includes chroma setting, hue setting, partial huechange, and color suppression.

After the various kinds of processing to the captured images under acertain exposure parameter, the images are subjected to distortioncorrection and image synthesis. A generated omnidirectional image isadded with a tag properly and stored in a file in the internal memory oran external storage. Inclination correction can be additionallyperformed on the basis of the information from the triaxial accelerationsensor 120 or a stored image file can be subjected to compression whenappropriate. A thumb-nail image can be generated by cropping or cuttingout the center area of an image.

In the above-described image processing the exposure parameter for thesolid-state image sensors 22A, 22B is determined and set in an exposurecondition register 200 by an exposure condition controller 210. Theimaging system 10 according to the present embodiment does not need toinclude a photometer for measuring the brightness of a subject but usesthe outputs of the solid-state image sensors 22A, 22B for exposurecontrol. To display a captured image on a LCD or an EVF (electronic viewfinder), image signals are constantly read from the solid-state imagesensors 22A, 22B. The exposure condition controller 210 repeatedlyconducts a photometry on the basis of a read image signal and determineswhether a brightness level is appropriate, to correct the exposureparameter such as F-value, exposure time (shutter speed), amplifier gain(ISO sensitivity) and obtain a proper exposure.

In omnidirectional photographing with the omnidirectional imaging system10, the two imaging units generate two images. In a photographic sceneincluding a high-brightness object as the sun, a flare may occur in oneof the images as shown in FIGS. 4A, 4B and spread over the entire imagefrom the high-brightness object. In such a case a synthetic image of thetwo images or omnidirectional image may be impaired in quality becausean increased offset of the one of the images causes a difference inbrightness at the connecting portions. Further, no proper object forexposure correction but an extremely white or black object will appearin an overlapping area of the two images.

In the imaging unit using fisheye lenses with total angle of view ofover 180 degrees, most of photographic areas do not overlap except forpartial overlapping areas. Because of this, it is difficult to acquire aproper exposure for the above scene by exposure correction based only onthe overlapping area. Further, even with the proper exposure obtainedfor the individual imaging units, a discontinuity of color attribute asbrightness may occur at the connecting positions of a synthetic image.

In view of avoiding insufficient exposure control, in the imaging system10 the exposure condition controller 210 is configured to evaluate thelevel of exposure of all of the images with the overlapping area andnon-overlapping areas of the images taken into consideration and decideexposure parameters as aperture a, exposure time t and amplifier gain gfor the solid-state image sensors 22A, 22B to be set in the exposurecondition register 200.

Specifically, the exposure condition controller 210 includes an areacalculator 212, an overall calculator 214 and an exposure conditiondeterminer 216, and can be realized by the ISPs 108 and CPU 130. In thefirst processing the ISPs 108A, 108B calculate the integrated value orintegrated average value for each divided area and outputs integrateddata for each divided area, and the exposure condition controller 210reads the integrated data.

FIGS. 5A, 5B show how to divide an image into small areas by way ofexample. In the present embodiment incident light on the lens systems20A, 20B is imaged on the light-receiving areas of the solid-state imagesensors 22A, 22B in accordance with a certain projection model such asequidistant projection. Images are captured on the two-dimensionalsolid-state area image sensors and image data represented in a planecoordinate system. In the present embodiment a circular fisheye lenshaving an image circle diameter smaller than an image diagonal line isused and an obtained image is a planar image including the entire imagecircle in which the photographic areas in FIGS. 4A, 4B are projected.

The entire image captured by each solid-state image sensor is dividedinto small areas in circular polar coordinate system with radius r andargument θ in FIG. 5A or small areas in planar orthogonal coordinatesystem with x and y coordinates in FIG. 5B. It is preferable to excludethe outside of the image circle from a subject of integration andaveraging since it is a non-exposed outside area. In the area divisionof the ISPs 108, each image is divided into small areas as shown inFIGS. 5A, 5B and the integrated value or integrated average value ofbrightness is calculated for each divided area. The integrated value isobtained by integrating the brightness values of all the pixels in eachdivided area while the integrated average value is obtained bynormalizing the integrated value with the size (number of pixels) ofeach divided area excluding the outside area.

The area calculator 212 receives the integrated data for each dividedarea including the integrated average value and calculates an indexvalue for each divided area to evaluate a photographic state thereof. Inthe present embodiment the index value is an area brightness level b forevaluating an absolute brightness of each divided area. The brightnesslevel b(x, y) of a certain divided area is calculated for eachsolid-state image sensor 22 by the following equation:

$\begin{matrix}{{b\left( {x,y} \right)} = {{s\left( {x,y} \right)}\frac{a^{2}}{t \cdot g}}} & (1)\end{matrix}$where s(x,y) is an integrated average value for a certain divided area,a is an aperture, t is exposure time, and g is amplifier gain. Thebrightness level b(r, θ) of a divided area in a circular coordinatesystem can be calculated in the same manner.

The brightness level b (x, y) is an index value to evaluate thebrightness of a subject in each divided area and calculated from thebrightness of a pixel value of an actual image according to a currentexposure parameter (a, t, g).

The overall calculator 214 evaluates the captured images including theoverlapping area as a whole on the basis of the calculated brightnesslevel b′(x, y) and calculates an overall evaluation value with weightingaccording to an overlapping portion between the photographic areas ofthe images. Herein, the solid-state image sensors 22A, 22B are referredto as 0^(th) and 1^(st) image sensors and their brightness levels arereferred to as b⁰ (x,y) and b¹(x,y), respectively. In the presentembodiment the overall evaluation value is an overall brightness levelbT to evaluate the brightness of all the areas of the images or subjectbrightness as a whole with a certain weighting.

The overall brightness level bT^(i) is calculated for an i-thsolid-state image sensor by the following equation:

$\begin{matrix}{{bT}^{i} = \frac{{\sum\limits_{x,y}{{b^{0}\left( {x,y} \right)} \times {w^{0\; i}\left( {x,y} \right)}}} + {\sum\limits_{x,y}{{b^{1}\left( {x,y} \right)} \times {w^{1\; i}\left( {x,y} \right)}}}}{{\sum\limits_{x,y}{w^{0\; i}\left( {x,y} \right)}} + {\sum\limits_{x,y}{w^{1\; i}\left( {x,y} \right)}}}} & (2)\end{matrix}$where b^(j) (x,y) is a brightness level for each divided area of eachsolid-state image sensor j (jε0, 1) and w^(ji) (x, y) is a weightedvalue in weighted averaging for each divided area.

As expressed by the above equation, different sets of weighted valuesw^(ji) (x, y) are used for each solid-state image sensor. The weightedvalues w^(ji) (x, y) can be adjusted so that a larger value is given toa solid-state image sensor with a lower brightness level to preventreceipt of an influence from a light source. Thus, weighting can beperformed properly in accordance with a result of determination about aphotographic scene. Further, the small areas (x, y) can be zoned intointermediate areas as overlapping area and non-overlapping area asindicated by hatching in FIGS. 5A, 5B. The overall brightness levelbT^(i) can be calculated by the following equation, using the brightnesslevel and weighted values for each intermediate area.

$\begin{matrix}{{bT}^{i} = \frac{{w\; 1\; i \times {bE}^{0}} + {w\; 2\; i \times {bE}^{1}} + {w\; 3\; i \times {bC}^{0}} + {w\; 4\; i \times {bC}^{1}}}{{w\; 1\; i} + {w\; 2\; i} + {w\; 3\; i} + {w\; 4\; i}}} & (3)\end{matrix}$where bE⁰ is a brighteness level (average) of an edge area (overlappingarea) of the 0^(th) image, bC⁰ is a brighteness level (average) of acenter area (non-overlapping area) of the 0^(th) image, bE¹ is abrightness level (average) of an edge area (overlapping area) of the1^(st) image, bC¹ is a brighteness level (average) of a center area(non-overlapping area) of the 1^(st) image, and w1 i to w4 i areweighted values of weighted averaging set for each intermediate area ofan i-th solid-state image sensor. The basic values of weighted values w1i to w4 i can be calculated by the following equations (4):W1i=W2i=Ae/(A0+A1+2Ae)W3i=A0/(A0+A1+2Ae)W4i=A1/(A0+A1+2Ae)where Ae is the size of the edge areas of the 0^(th) and 1^(st) images,and A0 and A1 are the sizes of the center areas of the 0^(th) and 1^(st)images. The calculated basis values can be corrected in accordance witha result of determination about a photographic scene such that asolid-state image sensor with a lower brightness level is given a largerweight.

Preferably, the overall calculator 214 can include a weighting setter toset the weighted values w^(ji)(x, y) according to a signal level of acaptured image. The weighting setter is configured to create abrightness distribution (histogram) from the brightness levels b^(j)(x,y) of all the divided areas and analyze a total average value andbrightness distribution for the scene determination. Then, according toa determined scene, it can change the weighted values w^(ji)(x, y) for acertain divided area depending on the brightness level b^(j)(x, y) forthe divided area in question.

For example, in a dark scene as night view, the weighting setter sets,by adding a predetermined amount to the area brightness level, theweighted value w^(ji)(x, y) to a larger value for evaluating a dividedarea with a larger brightness level b(x, y) calculated. Thereby, abright subject in a dark scene is highly evaluated for photometry andexposure can be properly controlled according to a result of thephotometry. Meanwhile, in a bright scene the weighting setter sets alarger weighted value w^(ji)(x, y) for a divided area with a smallerbrightness level b(x, y) calculated. Thus, a dark subject in a brightscene is highly evaluated for photometry.

Alternatively, a divided area containing an extremely black or whitesubject can be detected according to an upper limit threshold and alower limit threshold to exclude the divided area for a subject of thephotometry. For example, if a divided area or a white area with abrightness level equal to or over an upper limit threshold (b_(uth))and/or a divided area or a black area with a brightness level equal toor below a lower limit threshold (b_(lth)) is/are detected, these areascan be given a smaller weight or zero. Thereby, it is possible tocalculate the overall brightness level bT with the divided areaunsuitable for exposure correction given a small weight or not takeninto account.

The weighting setter determines a scene of each the 0^(th) and 1^(st)images captured by the two solid-state image sensors 22A, 22B on thebasis of a relation of the brightness between the two images and setsweighted values appropriate for the scene for each of the solid-stateimage sensors 22A, 22B. For example, to prevent receipt of an influencefrom a light source, the weighted values w^(ji) (x, y) can be adjustedso that a larger value is given to a solid-state image sensor with alower brightness level.

The exposure condition determiner 216 determines an exposure parameter(a, t, g) for each i-th solid-state image sensor on the basis of theoverall brightness level bT^(i) calculated by the overall calculator214. The overall brightness level bT is to evaluate the brightness of asubject in the images, and a condition for acquiring a proper exposurecan be represented by the following conditions:

$\begin{matrix}{\frac{a^{2}}{t} = \frac{{bT} \cdot g}{K}} & (4) \\{{{Bv} + {Sv}} = {{Av} + {Tv}}} & (5)\end{matrix}$where k is a constant, Bv is a brightness value, Sv is a sensitivityvalue, Av is an aperture value and Tv is a time value. The condition (5)is a transformation of the condition (4) taking 2 as a base of logarithmof each of the four parameters.

The four parameters are calculated by the following equations:

$\begin{matrix}{{Av} = {2\log_{2}a}} & (6) \\{{Tv} = {\log_{2}\frac{1}{t}}} & (7) \\{{Bv} = {\log_{2}\left( {k_{1}{bT}} \right)}} & (8) \\{{Sv} = {\log_{2}\left( {k_{2}g} \right)}} & (9)\end{matrix}$In the equations (8), (9) k₁, k₂ are constants.

Specifically, to satisfy the above exposure conditions, the exposurecondition determiner 216 adjusts the aperture a, exposure time t andamplifier gain g for each i-th solid-state image sensors 22 according toa current exposure parameter and a measured subject brightness oroverall brightness level and acquires a proper exposure. The correctedexposure parameter (a′, t′, g′) can be obtained from the brightnesslevel bT^(i), referring to a table called a program diagram which isprepared in advance in accordance with the characteristic of the imagingsystem 10. Herein, the program diagram refers to a diagram or a tablecontaining the combinations of the amplifier gain g and exposure time twith a fixed aperture a. Exposure values can be determined from thecombinations.

In the present embodiment an optimal combination of the aperture a,exposure time t and amplifier gain g is obtainable from the overallbrightness level bT by a certain program diagram. Alternatively, atleast one of the aperture a, exposure time t and amplifier gain g can bemanually set and the rest of them can be found from the program diagram.Such automatic exposure mode exemplifies shutter priority mode in whichexposure time t is manually set, aperture priority mode in whichaperture a is manually set, and sensitivity priority mode in whichamplifier gain g is manually set.

Hereinafter, the exposure control by the imaging system 10 is described,referring to FIGS. 6, 7. FIG. 6 is a flowchart for the exposure controlwhile FIG. 7 is a flowchart for exposure calculation process of theexposure control. The operation in FIG. 6 is repeatedly executed everytime images are captured by the image sensors 22A, 22B. In step S101 theimaging system 10 calculates an integrated average value for eachdivided area of the two image sensors 22A, 22B by integrating the pixelvalues thereof. In step S102 the exposure calculation in FIG. 7 iscalled up.

In FIG. 7 in step S201 the imaging system 10 takes the statistics ofintegrated average values for the divided areas of each solid-stateimage sensor 22 to calculate an average and a dispersion (or a standarddeviation) of the image. In step S202 the imaging system 10 determineswhether a current exposure parameter is in an allowable range from theaverage and dispersion of the two images to sufficiently evaluate asubject brightness. For example, if the average of the images is closeto zero and the dispersion is lower than a certain threshold, it isprobable that black saturation occur in the captured images. Incontrast, if the average is close to saturation and a dispersion is low,it is probable that white saturation occur in the captured images. Thelight amount of captured images with black or white saturation cannot beproperly measured so that the exposure parameter indicating black orwhite saturation is determined to be outside the allowable range.

With NO in step S202, the imaging system 10 proceeds to step S206 andadjusts an exposure parameter to acquire a proper exposure, andcompletes the operation in step S207. For instance, with the occurrenceof black saturation, the exposure parameter is adjusted so that theaperture is opened and exposure time and sensitivity are increased. Incontrast, with the occurrence of white saturation, the exposureparameter is adjusted so that the aperture is closed and exposure timeand sensitivity are decreased. When the gain g is fixed, the exposureparameter (a′, t′) is adjusted by lowering an exposure value Ev by apredetermined number of steps in black saturation. In white saturationthe exposure parameter (a′, t′) is adjusted by raising an exposure valueEv by a predetermined number of steps. When the aperture a is fixed, theexposure parameter (t′, g′) is adjusted by increasing the exposure timet and amplifier gain g in black saturation and decreasing them in whitesaturation.

With YES in step S202, the area calculator 212 calculates a brightnesslevel b^(i)(x,y) for each divided area of the 0^(th) and 1^(st) imageson the basis of the integrated average value s(x, y) and a currentexposure parameter (a, t, g).

In step S204 the overall calculator 214 determines a scene of the imagesand reads a weighted value w^(ji)(x, y) for the determined scene. Instep S205 the overall calculator 214 calculates the weighted average ofthe brightness levels b⁰(x, y) and b¹(x, y) and calculates the overallbrightness level bT^(i) for each i-th solid-state image sensors by theequations (2) and (3). In step S206 the exposure condition determiner216 adjusts the exposure according to the overall brightness levelbT^(i) to satisfy the conditions (4) and (5) and determines the exposureparameter (a′, t′, g′). Then, the imaging system 10 completes theexposure calculation and returns to step S103 in FIG. 6.

In step S103 the exposure parameter in the exposure condition register200 is updated to the determined exposure parameter (a′, t′, g′),completing the exposure control operation. By repeating the operationsin FIGS. 6, 7, the exposure condition is set to a proper exposuresatisfying the above conditions (4), (5).

In omnidirectional photographing with the omnidirectional imaging system10, the same subject is captured in the overlapping area, therefore, thebrightness levels b of the two images should be the same value. In viewof the occurrence of flares in one of the images as shown in FIGS. 4A,4B, according to the present embodiment the overall exposure level ofthe captured images including the overlapping area and non-overlappingareas is evaluated to determine the exposure parameter (a, t, g) foreach of the imaging units. Thereby, it is made possible to abate adiscontinuity of the brightness at the connecting positions of asynthetic image and generate high-quality synthetic images.

Thus, according to the above embodiment, it is possible to provide animaging controller and imaging control method and program which canprovide to each of imaging units a proper imaging condition to abate adiscontinuity at the connecting points of the images captured by theimaging units in synthesizing the images.

The above embodiment has described an example where two images capturedwith the lens systems having angle of view of over 180 degrees areoverlapped for synthesis. Alternatively, three or more images capturedwith multiple imaging units can be overlapped for synthesis.

Moreover, the above embodiment has described the imaging system 10 tocapture an omnidirectional still image as an example of the imagingcontroller. The present invention should not be limited to such anexample. Alternatively, the imaging controller can be configured as anomnidirectional video imaging system or unit, a portable data terminalsuch as a smart phone or tablet having an omnidirectional still or videoshooting function, or a digital still camera processor or a controllerto control a camera unit of an imaging system.

The functions of the omnidirectional imaging system can be realized by acomputer-executable program written in legacy programming language suchas assembler, C, C++, C#, JAVA® or object-oriented programming language.Such a program can be stored in a storage medium such as ROM, EEPROM,EPROM, flash memory, flexible disc, CD-ROM, CD-RW, DVD-ROM, DVD-RAM,DVD-RW, blue ray disc, SD card, or MO and distributed through anelectric communication line. Further, a part or all of the abovefunctions can be implemented on, for example, a programmable device (PD)as field programmable gate array (FPGA) or implemented as applicationspecific integrated circuit (ASIC). To realize the functions on the PD,circuit configuration data as bit stream data and data written in HDL(hardware description language), VHDL (very high speed integratedcircuits hardware description language), and Verilog-HDL stored in astorage medium can be distributed.

Although the present invention has been described in terms of exemplaryembodiments, it is not limited thereto. It should be appreciated thatvariations or modifications may be made in the embodiments described bypersons skilled in the art without departing from the scope of thepresent invention as defined by the following claims.

The invention claimed is:
 1. An imaging controller comprising: an indexcalculator to calculate an index value, which is a brightness level toevaluate a brightness, for each of divided areas of images captured by aplurality of imaging units, the index value for evaluating aphotographic state of each of the divided areas; an evaluation valuecalculator to evaluate the images and an overlapping area between theimages on the basis of the index value of each divided area calculatedby the index calculator and to calculate brightness level for each ofthe imaging units to evaluate a total brightness of the images byapplying weighting to the overlapping area; and a condition determinerto determine an imaging condition including an exposure condition foreach of the imaging units on the basis of the brightness level for eachof the imaging units calculated by the evaluation value calculator. 2.The imaging controller according to claim 1, further comprising a setterto set, for each of the imaging units, a set of weighted values for thedivided areas according to a relation of brightness between the imagescaptured by the imaging units.
 3. The imaging controller according toclaim 1, further comprising a weighting setter to apply weighting to acertain divided area in accordance with the index value calculated forthe certain divided area.
 4. The imaging controller according to claim3, wherein the weighting setter is configured to apply weighting to thedivided areas such that in a dark scene a divided area for which alarger index value is calculated is given a larger weighting.
 5. Theimaging controller according to claim 3, wherein the weighting setter isconfigured to apply weighting to the divided areas such that in a brightscene a divided area for which a smaller index value is calculated isgiven a larger weighting.
 6. The imaging controller according to claim3, wherein the weighting setter is configured to apply weighting to thedivided areas such that a divided area for which the index value as anupper limit threshold or more or a lower limit threshold or less iscalculated is given a smaller weighting or zero.
 7. The imagingcontroller according to claim 1, wherein the imaging condition includesat least one of an aperture, an exposure time and a gain.
 8. An imagingcontrol method comprising the steps of: calculating an index value,which is as brightness level to evaluate a brightness, for each ofdivided areas of images captured by a plurality of imaging units, theindex value for evaluating a photographic state of each of the dividedareas; evaluating the images and an overlapping area between the imageson the basis of the index value of each divided area calculated in thecalculating step; calculating a brightness level for each of the imagingunits to evaluate a total brightness of the images by applying weightingto the overlapping area; and determining an imaging condition includingan exposure condition for each of the imaging units on the basis of thebrightness level for each of the imaging units calculated in theevaluating step.
 9. A non-transitory computer-readable storage mediumstoring computer executable instructions which, when executed by acomputer, cause the computer to: calculate an index value, which is asbrightness level to evaluate a brightness, for each of divided areas ofimages captured by a plurality of imaging units, the index value forevaluating a photographic state of each of the divided areas; evaluatethe images and an overlapping area between the images on the basis ofthe index value of each divided area; calculate a brightness level foreach of the imaging units to evaluate a total brightness of the imagesby applying weighting to the overlapping area; and determine an imagingcondition including an exposure condition for each of the imaging unitson the basis of the brightness level for each of the imaging units.